Method and system for activity classification

ABSTRACT

A method and system for activity classification. A pressure sensor receives input data resulting from physical activity of a subject performing an activity. The input data includes pressure data from at least one pressure sensor, and may include other data acquired through other types of sensors. A deep learning neural network is applied to the input data for identifying the activity. The neural network is trained with reference to training data from a training database. The training data may include empirical data from a database of previous data of corresponding activities, synthesized data prepared from the empirical data or simulated data. The training data may include data from physical activity of the subject being monitored by the system. Different aspects of the neural network may be trained with reference to the training data, and some aspects may be locked or opened depending on the application and the circumstances.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/640,765 having a national entry date of Feb. 21, 2020; which is anational entry of PCT/CA2018/051011 filed Aug. 22, 2018; which claimsthe benefit of priority of U.S. Patent Application No. 62/548,676 filedAug. 22, 2017. All of the above applications are incorporated herein byreference.

FIELD

The present disclosure relates to classification of physical activities.

BACKGROUND

Monitoring and quantifying physical activity facilitates effectiveexercise and healthy living. Devices that count steps and log exerciseare increasingly common as people increasingly take ownership of andinterest in their own well-being. Similarly, careful monitoring ofactivity on the worksite can be important in preventingworkplace-related injuries. However, even with the most precise of theseavailable devices, distinguishing between certain types of activity canbe very difficult.

Commonly used devices apply inertial data, primarily from accelerometersand gyroscopes, to determine activity type, though several technologiesalso include pressure sensing and acoustic measurement systems. Activitytype is then classified by comparing data collected from one or more ofthese sensors to predetermined, pre-programmed characteristics of anumber of common activities. However, these systems are limited in theirability to distinguish between unique occurrences of similar activities,to classify transition stages between definitive activity types, or toaccount for personal attributes of the subject being monitored.

There is call for improved approaches to classification of physicalactivities that can distinguish between potentially small measurabledifferences between activities, and accurately classify physicalactivities.

SUMMARY

Herein provided are a system and method for classifying physicalactivity of a subject. Input data including pressure data is collectedfrom sensors during activities performed by the subject. The input datamay also include other types of data. A deep learning neural network(“NN”) is applied to the input data, providing classified activity datadefining the activities. Weights and biases define relationships betweenconsecutive layers within the NN. The weights and biases may beinitially set and updated by supervised training to account for specificattributes of the subject, facilitating activity classification for thesubject specifically. The supervised training may include populationtraining based on training data from a training database, which may beused in a network training scenario. The supervised training may includepersonalized subject training based on input data from an individual.The individual may be the subject or an individual other than thesubject. Input data used in training may be from a previously recordedactivity session and applied to training using batch processing, whichmay facilitate training of the system prior to use. Once use has begun,the input data may be streamed directly from the sensor module,requiring online or near-real-time processing. The real-time training oractivity classification may be applied after training of the system.Certain layers and nodal relationships within the NN may be locked aftercalibration by the training data, and other layers and nodalrelationships updated with ongoing training.

The NN may apply activity classification across an entire data set ofthe input data. The NN may apply activity classification across aspecified window of time of the input data. The window of time may bethe known and static width of an event identified by a user, who may bethe subject or a different individual, within the input data, in whichthe event is a distinct waveform within the input data and delineatedusing known event detection methods. The window of time may be of adefined time range that is progressed through the input data and theinput data within the time window is classified before translating thetime window along a timeline of the input data to apply the NN to asubsequent portion of the input data within the time window aftertranslation. The time window may be weighted, such that it has aweighted segment in which data within that segment are given a weightedpreference when the NN is applied to classification of the activitywithin the bounds of the weighted time window. The input data may alsobe classified through an event window in which an event is detected inthe input data, event limits defined on either side of the event, andthen the input data within the event window may be classified. Theclassified activity data may be communicated to a user by display of theclassified activity or by storage of the classified activity data. Theuser may be the subject or an individual who is not the subject. Thesubject may be the user, an individual who is not the user, a prostheticlimb used by an individual, an animal, an autonomous vehicle, a robot,or any suitable subject that may apply pressure to a surface duringactivity.

The pressure data may be collected from pressure sensors located on thebody of the subject. Pressure sensors may be placed on the plantarportion of the foot to measure plantar pressure during physicalactivity. Sensors for other types of input data indicative of a physicalactivity may also be used in the system. The input data may also becollected from a variety of sensors, including accelerometer, agyroscope, a seismograph, a thermometer, or a humidity sensor, analtimeter, a GPS, a video camera, a heart rate sensor, an oxygen sensor,a breathing rate sensor, a blood glucose sensor, a fatigue measuringdevice, a limb position measuring device, a blood pressure monitor, anECG, a lung function meter, an alcohol level sensor, drug level sensor,or any other type of sensor that measures the level of impairment, orany other sensor that receives the input data.

The NN may be a neural network with translational invariance. The NN maybe a convolutional neural network (“CNN”). The CNN includes at least oneconvolutional layer and at least one fully connected layer. The NN maycomprise long short-term memory (“LSTM”) units. The NN may be acombination of neural network types, for example, using a CNN to modelrelationships between spatial components of the system, such asdifferent sensors, and using an LSTM system to model temporal componentsof the system, such as data points in time.

Population training includes providing population training data to theNN. The population training data includes pressure data and may includeother input data acquired from a variety of subjects with varyingphysical attributes and performing one or more defined activities. Thevariety of subjects makes up a population. Input data resulting fromeach of the subjects in the population performing the defined activitiesmakes up a training database. The training database may includeempirical input data or synthetic input data generated from pressuredata or other input data. The NN classifies activities based on thepopulation training data to provide classified training data. Theclassifications may be in the form of probabilities of likelihood that agiven estimated activity is the actual activity. The classifications maybe expressed as the likelihood that a given portion of the classifiedactivity data corresponds to is the actual activity. The actualactivities and the corresponding classified actual activity data areknown. The classified actual activity data are the known classificationof the actual activities. The classified actual activity data isprovided through a training confirmation and the classified actualactivity data is compared with the classified training data. The NN maybe updated according to a loss function defined as the measure ofdiscrepancy between the classified training data and the classifiedactual activity data by adjusting the weights and biases to mitigate theloss function. The measure of discrepancy may be the difference, thesquared difference, cross-entropy, or another measure of discrepancybetween data sets. Population training data including a variety ofactivities performed by a variety of subjects may be accessed from thetraining database. The NN may be applied to the population training dataand the weights and biases updated based on the classified activity dataresulting from applying the NN to the training database. Storedclassified activity data may be combined with the population trainingdata and included in the training database for facilitating populationtraining for the subject and for other subjects who access the trainingdatabase. The population training may be applied initially to optimizethe weights and biases before use, and may be repeated thereafter,including by receiving updated training data periodically orsporadically.

Subject training may be applied to train the NN with reference tosubject training data. The subject training data may include input dataresulting from the subject performing one or more defined activities.The NN classifies the defined activity based on the pressure data toprovide the classified activity data. The classifications may beexpressed as a probability that the classified activity data correspondsto the actual activity. The defined activity is known and thecorresponding classified actual activity data that is expected to resultfrom application of the NN to the pressure data associated with thedefined activity are known. The classified actual activity data iscompared with the classified activity data. The NN may be updatedaccording to a loss function defined between the classified activitydata and the classified actual activity data by adjusting the weightsand biases to mitigate the loss function. The subject's specificattributes direct the resulting input data, and as a result, the subjecttraining is based on data that is inherently specific to the subject.Subject training may be applied in association with a confirmation inputthat provides the classified actual activity data. Subject training maybe applied following a prompt to the subject to perform the definedactivity. Subject training may be a specific routine that the NN appliesin response to a user input, or may be available during the ordinarycourse of activity classification in response to the confirmation inputor following the prompt.

The subject training data may be used to train the NN after it hasalready be trained by the training database. Once the NN has beentrained by the training database, providing the population-trained NN,the weights and biases of multiple layers of the NN are defined. Theweights and biases within all or a subset of the layers of the NN may belocked, such that the subject training data cannot change the weightsand biases. Administration of the subject training data to thepopulation-trained NN results in loss functions when compared to theclassified actual activity data corresponding to the subject trainingdata. The NN may be updated according to the loss function by adjustingthe weights and biases that are not locked to minimize the lossfunction, or the system loss function. Subject training may be appliedto adjust the weights and biases that are not locked within the NN,providing the personalized NN that is based on a population-trained NN.The weights and biases that are not locked may be within subsequentlayers of the NN that are downstream of an initial input layer. Forexample, subject training of a CNN may adjust the weights and biaseswithin a fully connected layer and a pre-classification layer, whileconvolutional layers may be locked during population training.

Processing of the classified activity data through the NN, thepopulation training or the subject training may be applied afterinputting user data with an input module. The user data may definespecific attributes of the subject and otherwise relate to the subject.Where the subject is an individual, user data may include personalattributes of the subject (e.g. age, sex, weight, height, shoe size,physical conditions, mental conditions, limb length, body fat index,location, elevation, limb position, number of hours awake, or anyattributes that relate to the user, etc.). The user data may be appliedto selecting initial weights and biases, or updating the weights andbiases, based on the training database or the subject training data.Where a CNN is used, the user data may be concatenated into the fullyconnected layer for classifying the convoluted and pooled input data. Inan NN other than a CNN, the user data may be input into one of thelayers of the NN. Population training or subject training may facilitateapplication of the NN for classifying activity performed by the subjectover a session of a few seconds by taking into account specificattributes of the subject from the user data. If the subject changesactivities during the session, the NN may classify multiple activitiestypes performed during the session. The NN may classify transitions fromone activity type to the next, and between separate occurrences of oneactivity type.

Processing of the input data through the NN may be in real-time duringcollection of data, or in a batch after data has been collected. A timewindow defining a time range may be applied to the input data and the NNapplied to the input data within the time window. The NN may be appliedto the input data corresponding to the time points within the timewindow and the input data classified, providing classified activitydata. The time window is then progressed along a timeline of the inputdata by a defined amount, and the data within that window is inputted tothe NN and classified, providing a rolling account of estimations of theactivities performed during acquisition of the input data. The inputdata within each time window may be weighted such that a scaling valuethat increases the importance of a portion of the input data within thetime window.

Processing of the input data through the NN may be done for onlyspecific sections of the input data that have been identified using aknown event detection method. The event detection method may delineatedistinct events within the data set, and the data within the delineatedevent may be input into the NN and classified, providing classifiedactivity data.

The classified activity data may be presented as a composition ofestimated activities, such that the classified activity data ispresented as a percentage breakdown of a variety of activities. Thehighest percentage activity from within the composition of estimatedactivities represents the activity that the NN has identified. Thecomposition of estimated activities may represent a transition betweenactivities. For example, the NN may classify the activity with an 80%chance of being walking, and 20% chance of being running, which mayrepresent a transition between walking and running.

The activity classification system may be applied to measurement ofpressure on the soles of a subject to classify activities that involvegait, steps and other activities that result in pressure on the soles ofthe subject. The activity classification system may be applied tomeasurement of pressure and blood glucose levels in cascade to correlateactivity levels with blood glucose, and in some cases to deliver insulinin response to blood glucose levels. Similarly, indicators for the needto administer drugs, therapeutics or other substances based on theclassified activity data may be leveraged to prompt the subject or acaregiver to provide the substance, or the indicators may trigger anautomatic delivery system. The activity classification system may beapplied to fall prediction by measurement of pressure on soles topredict an imminent fall and warn the subject. Systems directed to falldetection may also contact caregivers if a fall is detected, and mayinclude inertial sensors to facilitate characterizing aspects of gaitrelevant to slipping and falling. Such systems may alert the subjectthrough tactile or other intuitive feedback that allows an instinctiveresponse to the feedback. Similarly, the system may be applied toprompting users.

In a first aspect, herein provided is a method and system for activityclassification. A pressure sensor receives input data resulting fromphysical activity of a subject performing an activity. The input dataincludes pressure data from at least one pressure sensor, and mayinclude other data acquired through other types of sensors. A deeplearning neural network is applied to the input data for identifying theactivity. The neural network is trained with reference to training datafrom a training database. The training data may include empirical datafrom a database of previous data of corresponding activities,synthesized data prepared from the empirical data or simulated data. Thetraining data may include data from physical activity of the subjectbeing monitored by the system. Different aspects of the neural networkmay be trained with reference to the training data, and some aspects maybe locked or opened depending on the application and the circumstances.

In a further aspect, herein provided is a method for classifying anactivity of a subject comprising: receiving input data of the subjectresulting from the activity, the input data including pressure data;applying a deep learning neural network to the input data based onweights and biases, resulting in classified activity data; training thedeep learning neural network for updating the weights and biases; andcommunicating the classified activity data to a user.

In some embodiments, the input data comprises binary inputs of dataabove or below a threshold value of the input data.

In some embodiments, the input data comprises quantitative inputs of apressure magnitude detected by a pressure sensor.

In some embodiments, the input data comprises additional data.

In some embodiments, the additional data comprises data from anaccelerometer, a gyroscope, a seismograph, a thermometer, or a humiditysensor, an altimeter, a GPS, a video camera, a heart rate sensor, anoxygen sensor, a breathing rate sensor, a blood glucose sensor, afatigue measuring device, a limb position measuring device, a bloodpressure monitor, an ECG, a lung function meter, an alcohol levelsensor, or drug level sensor.

In some embodiments, the weights and biases are initially determinedwith reference to relationships between separate sources of the inputdata.

In some embodiments, the separate sources of the input data compriseseparate sensors on a support matrix located to receive the input datafrom different portions of the subject; and the weights and biases areinitially determined with reference to the locations of the separatesensors on the support matrix.

In some embodiments, training the deep learning neural networkcomprises: confirming the activity, resulting in a defined activity andcorresponding classified actual activity data; defining a loss functionbetween the classified actual activity data and the classified activitydata; and updating the weights and biases for mitigating the lossfunction.

In some embodiments, confirming the activity comprises prompting thesubject to perform the defined activity.

In some embodiments, confirming the activity comprises receiving aconfirmation input that the subject performed the defined activity.

In some embodiments, receiving the confirmation input comprisesreceiving the confirmation input from the subject.

In some embodiments, receiving the confirmation input comprisesreceiving the confirmation input from an individual other than thesubject.

In some embodiments, the deep learning neural network comprises atranslationally invariant neural network.

In some embodiments, the translationally invariant neural networkcomprises: at least one convolutional layer; and at least one fullyconnected layer subsequent to the at least one convolutional layer.

In some embodiments, the method includes receiving user data ofattributes of the subject; and wherein the fully-connected layercomprises the user data concatenated with the fully-connected layer.

In some embodiments, the translationally invariant neural networkcomprises a long short-term memory system.

In some embodiments, training the deep learning neural networkcomprises: receiving training data of at least one reference subjectperforming a defined activity from a training database; applying thedeep learning neural network to the training data based on the weightsand biases, resulting in classified training data; receiving classifiedactual activity data from the training database; defining a lossfunction between the classified training data and the classified actualactivity data; and updating the weights and biases for mitigating theloss function.

In some embodiments, the training data comprises empirical input data.

In some embodiments, the training data comprises synthetic input datagenerated from the empirical input data.

In some embodiments, the synthetic input data is generated from theempirical input data by applying to the empirical input data one or manyof the following operations: time-shifting, magnitude-scaling andspectral magnitude-scaling.

In some embodiments, the reference subject is the subject.

In some embodiments, the method includes receiving a confirmation inputthat the subject performed the defined activity and wherein theempirical input data comprises input data confirmed to correspond to thedefined activity by the confirmation input.

In some embodiments, the empirical input data comprises input datareceived following a prompt to the subject to perform the definedactivity.

In some embodiments, the reference subject comprises an individualselected with reference to attributes of the subject.

In some embodiments, the reference subject comprises a plurality ofindividuals.

In some embodiments, the training data comprises data of the pluralityof individuals performing the defined activity, and at least a portionof the plurality of individuals performing at least one activity otherthan the defined activity.

In some embodiments, training the deep learning neural networkcomprises: initial training based on first relationships between a firstgroup of layers of the deep learning neural network; locking the firstrelationships; and subsequent training based on second relationshipsbetween a second group of layers of the deep learning neural network.

In some embodiments, the initial training is based on training data in atraining database.

In some embodiments, the subsequent training is based on updatedtraining data received from the training database.

In some embodiments, the subsequent training is based on empirical dataof the subject.

In some embodiments, communicating the classified activity data to theuser comprises displaying the classified activity data, providingtactile stimulus to the subject, or storing the classified activity in adatabase.

In some embodiments, the activity comprises activity indicative of animminent fall and communicating the classified activity data to the usercomprises a tactile or other neuroplastic stimulus to prompt the user tocorrect the activity and avoid a fall.

In some embodiments, the activity comprises activity indicative of animminent fall and communicating the classified activity data to the usercomprises a tactile or other neuroplastic stimulus to prompt the user tocorrect the activity and avoid a fall.

In some embodiments, the activity further comprises a fall and furthercomprising communicating the classified activity data of the fall to athird party.

In some embodiments, the method includes applying a time window to theinput data to provide time-segmented input data; and wherein applyingthe deep learning neural network to the input data comprises applyingthe deep learning neural network to the time-segmented input data.

In some embodiments, the method includes weighting the time-segmentedinput data to provide weighted input data and wherein applying the deeplearning neural network to the input data comprises applying the deeplearning neural network to the weighted input data.

In some embodiments, applying an event detection filter to the inputdata to event-classified input data; and wherein applying the deeplearning neural network to the input data comprises applying the deeplearning neural network to the event-classified input data.

In some embodiments, the method includes receiving user data ofattributes of the subject.

In some embodiments, the user data includes medical data of the subject.

In some embodiments, the weights and biases are updated with referenceto the user data.

In some embodiments, the method includes receiving metabolic load data;applying the classified activity data and the metabolic load data to ametabolic load neural network, providing classified combined metabolicand activity data; and communicating the classified combined metabolicand activity data to the user.

In some embodiments, the method includes administering a substance tothe subject in response to the classified combined metabolic andactivity data.

In some embodiments, the method includes applying the classifiedactivity data to a second deep learning neural network, resulting insecond classified activity data.

In some embodiments, the method includes receiving second input data,and wherein applying the classified activity data to a second deeplearning neural network comprises applying the second input data and theclassified activity data to the second deep learning neural network.

In some embodiments, the activity includes standing, sitting, lying,crouching, walking, running, shuffling, skipping, dancing, ascending ordescending stairs, a ramp or a ladder, cycling, situps, crunches,pushups, skiing, snowboarding, lifting weights, deadlift, lunges,jumping jacks, squats, mountain climbers, bench dips, step ups,planking, calf raises, burpees, growing, shoulder press, bicep curls,tricep dips, speed-walking, skateboarding, bowling, driving, tappingfeet or otherwise fidgeting, or other steps or footfalls of the subject,which may change speed, pace, or direction.

In a further aspect, herein provided is a system for classifyingactivity of a subject comprising: a sensor module comprising a pressuresensor for receiving input data during the activity, the input dataincluding pressure data; a processor configured for receiving the inputdata, the processor configured for executing a method comprising:applying a deep learning neural network to the input data based onweights and biases, resulting in classified activity data; and trainingthe neural network for updating the weights and biases.

In some embodiments, the sensor module comprises at least two pressuresensors; and the weights and biases are biased based on knownrelationships between the at least two pressure sensors.

In some embodiments, the input data comprises additional data.

In some embodiments, the sensor module comprises an accelerometer, agyroscope, a seismograph, a thermometer, or a humidity sensor, analtimeter, a GPS, a video camera, a heart rate sensor, an oxygen sensor,a breathing rate sensor, a blood glucose sensor, a fatigue measuringdevice, a limb position measuring device, a blood pressure monitor, anECG, a lung function meter, an alcohol level sensor, drug level sensor,or any other type of sensor that measures the level of impairment forreceiving the additional data.

In some embodiments, the sensor module comprises a sensor for receivinga magnitude of input data.

In some embodiments, the sensor module comprises a sensor for receivingan indication of whether a threshold of the input data has beenexceeded.

In some embodiments, the deep learning neural network comprises atranslationally invariant neural network.

In some embodiments, the neural network comprises: at least oneconvolutional layer; and at least one fully-connected layer subsequentto the at least one convolutional layer.

In some embodiments, the fully-connected layer comprises user dataconcatenated with the fully-connected layer.

In some embodiments, applying a deep learning neural network to theinput data comprises applying a time window to the input data to providetime-segmented input data; and applying the deep learning neural networkto the time-segmented input data.

In some embodiments, the system includes weighting the time-segmentedinput data to provide weighted input data and wherein applying the deeplearning neural network to the input data comprises applying the deeplearning neural network to the weighted input data.

In some embodiments, applying a deep learning neural network to theinput data comprises applying an event detection filter to the inputdata to event-classified input data; and applying the deep learningneural network to the event-classified input data.

In some embodiments, training the deep learning neural networkcomprises: confirming the activity, resulting in a defined activity andcorresponding classified actual activity data; defining a loss functionbetween the classified actual activity data and the classified activitydata; and updating the weights and biases for mitigating the lossfunction.

In some embodiments, the training data comprises synthetic input datagenerated from empirical input data.

In some embodiments, training the deep learning neural network comprisesinitial training based on first relationships between a first group oflayers of the deep learning neural network; locking the firstrelationships; and subsequent training based on second relationshipsbetween a second group of layers of the deep learning neural network.

In some embodiments, the system applies any of the embodiment of themethod described herein.

In some embodiments, the activity includes standing, sitting, lying,crouching, walking, running, shuffling, skipping, dancing, ascending ordescending stairs, a ramp or a ladder, cycling, situps, crunches,pushups, skiing, snowboarding, lifting weights, deadlift, lunges,jumping jacks, squats, mountain climbers, bench dips, step ups,planking, calf raises, burpees, growing, shoulder press, bicep curls,tricep dips, speed-walking, skateboarding, bowling, driving, tappingfeet or otherwise fidgeting, or other steps or footfalls of the subject,which may change speed, pace, or direction.

Other aspects and features of the present disclosure will becomeapparent to those ordinarily skilled in the art upon review of thefollowing description of specific embodiments in conjunction with theaccompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present disclosure will now be described, by way ofexample only, with reference to the attached figures, in which referencenumerals having a common final two digits refer to correspondingfeatures across figures (e.g. the system 10, 110, 210, 310, 410, 510,610, 710, etc.):

FIG. 1 is a block diagram illustrating selected hardware and processcomponents of an activity classification system using a CNN;

FIG. 2 is a schematic of the system of FIG. 1 configured for receivingpressure data along the sole of a user;

FIG. 3 is a schematic of a CNN used in the system of FIG. 1 ;

FIG. 4 is a flowchart of a method for classifying physical activities ofa user with the CNN of FIG. 3 and training the CNN;

FIG. 5 is a plot of example pressure data acquired with the system ofFIG. 1 being processed with a time window;

FIG. 6 is a schematic of a system for receiving pressure data along thesole of a subject;

FIG. 7 is a schematic of a CNN used in the system of FIG. 6 ;

FIG. 8 is a schematic of a system for receiving pressure data along thesole of a subject;

FIG. 9 is a block diagram illustrating selected hardware and processcomponents of an activity classification system using a NN;

FIG. 10 is a flowchart of a method for classifying physical activitiesof a user with the NN of FIG. 9 and training the NN;

FIG. 11 is a plot of example pressure data acquired with the system ofFIG. 9 being processed with a weighted time window;

FIG. 12 is a schematic of a system for receiving pressure data along thesole of a subject and receiving blood glucose data from the subject;

FIG. 13 is a block diagram of a system for classifying physicalactivities of a user with an activity classification NN and a metabolicload NN using the system of FIG. 12 ;

FIG. 14 is a system for receiving pressure data along the sole of asubject, receiving blood glucose data from the subject and administeringinsulin in response to the blood glucose data;

FIG. 15 is a block diagram of a system for classifying physicalactivities of a user with an activity classification NN cascaded ametabolic load NN, and administering insulin using the system of FIG. 14;

FIG. 16 is an activity classification system for fall prevention using aNN;

FIG. 17 is a flowchart of a method for classifying physical activitiesof a user with a NN and training the NN;

FIG. 18 is a schematic of example pressure data acquired with the systemof FIG. 16 being processed with an event detection system; and

FIG. 19 is a schematic of a method using a first NN to provideclassified activity data as input data to a second NN.

DETAILED DESCRIPTION

Herein provided are a system and method for classifying physicalactivity of a subject based on input data resulting from the subject'sphysical activity. The input data includes pressure data collected fromsensors during activities performed by the subject. Additional sensorsmay be applied to provide other types of data to complement the pressuredata (e.g. an accelerometer, a gyroscope, a seismograph, a thermometer,or a humidity sensor, an altimeter, a GPS, a video camera, a heart ratesensor, an oxygen sensor, a breathing rate sensor, a blood glucosesensor, a fatigue measuring device, a limb position measuring device, ablood pressure monitor, an ECG, a lung function meter, an alcohol levelsensor, drug level sensor, or any other type of sensor that measures thelevel of impairment, etc.). The subject may be an individual, aprosthetic limb used by an individual, an animal, an autonomous vehicle,a robot, or any suitable subject that may apply pressure to a surfaceduring activity.

A deep learning neural network (“NN”) is applied to input data includingthe pressure data and any additional types of data, providing classifiedactivity data defining the activities of the subject. The NN includeslayers made of nodes. Weights and biases define relationships betweenthe nodes of an NN. The NN may include a convolutional neural network(“CNN”), a long short-term memory (“LSTM”) system, or any other NN thatincorporates translational invariance. The NN is a deep learningartificial intelligence engine. Where a CNN is applied, the CNN includesat least one convolutional layer and at least one fully connected layer.Known relationships between inputs of input data are used to defineweights and biases connecting nodes in consecutive layers within the NNto one another. The NN classifies activities based on inputs of theinput data according to weights and biases in each layer. The weightsand biases may be initially defined with reference to random valueswithin a defined range, previous data in a training database, userinputs to the NN, the relationships between input data points fromseparate moments in time. The weights and biases may be initiallydefined by relationships between nodes, and may be related in a varietyof ways. The nodes that represent different sensors in space may berelated to one another for an instant in time. For example, pressuresensors on heels of each of a pair of may be instantiated as negativelycorrelated, whereas two toe sensors on the same foot might have positivecorrelation. In another example, nodes that represent different timepoints for one sensor may have relationships for nodes that are close intime, such as a correlation between a current value and a value read 10milliseconds ago, in which case a high reading may climb then faderather than jumping instantaneously from low to high. Definition ofnodes being close in space or in time would be driven by the particularuse case.

The weights and biases may be updated by supervised training of the NNbased on population training. The weights and biases may be updated bysupervised training of the NN based on subject training to recognize thespecific attributes of the subject, facilitating activity classificationfor the subject specifically. The supervised training may includesubject training based on the input data or population training based ontraining data from a training database.

The training data may be empirical data, synthetic data based onempirical data or simulated data. The classified activity data may becommunicated to a user by displaying or storing the classified activitydata. The user may be the subject or an individual who is not thesubject. The training database may be built from training data from oneor more subjects, one or more experts, or combinations thereof. Thetraining database may serve as a coaching tool to classify the subject'sactivities to the subject's past activities. The training database mayserve as a coaching tool to classify the subject's activities inrelation to an expert or to a set of users.

In the NN, the input data may be plotted as a two-dimensional matrix.The matrix represents a window subset of pressure data from eachpressure sensor, any sources of sensor data other than pressure data,any manual input from the user, and any other sources of the input data.The matrix may be represented as rows and columns. Each pressure sensoror other source of the input data may be represented as a column ofdata. Each row may be represented as a moment in time, with dataextending along a column representing data from one sensor or othersource of data acquired over time. Each cell corresponds to a data node.Connections between nodes in convolutional layers may define patterns inthe overall data series. The connections defined between nodes providethe basis for the NN to define and identify activities. The moredeveloped the connections between nodes, the greater degree to which theNN may facilitate activity classification.

Connections may be made between nodes in the pressure data matrix andnodes in a feature map of a subsequent convolutional layer. Connectionsmay be made between nodes in feature maps of adjacent convolutionallayers. Connections may be made between nodes of feature map of aconvolutional layer and a fully-connected layer. Connections betweennodes may be forgone if it is determined that there is no relationshipbetween the points.

Connections may be made between nodes in the pressure data matrix and afeature map of a convolutional layer based on data in the pressure datamatrix from a selected pressure sensor or other source of the input dataacross a span of time (i.e. along columns in the two-dimensionalmatrix). Connections may be made between nodes in the pressure datamatrix and a feature map of a convolutional layer based on data in thepressure data matrix from different pressure sensors or other sources ofthe input data that represent data from different pressure sensors orother sources of the input data in a single moment in time (i.e. alongrows in the two-dimensional matrix).

A pressure sensor or other sources of input data may provide the inputdata in predictable patterns according to the activity being performedby the user. Where patterns in the input data from a pressure sensor orother sources of input are known, minimally correlated patterns aregiven low values, highly correlated patterns are given large positivevalues, and negatively correlated patterns are given large negativevalues.

Separate pressure sensors or other sources of input data may havepredictable relationships with one another according to the activitybeing performed by the user. Where relationships between pressuresensors, or between pressure sensors and other sources of input areknown, relationships between minimally correlated pressure sensors orother sources of input are given low values, relationships betweenhighly correlated pressure sensors or other sources of input are givenlarge positive values, and relationships between negatively correlatedpressure sensors or other sources of input are given large negativevalues.

The method and system may be applied to activity classification of asubject undertaking a variety of physical activities (e.g. standing,sitting, lying, crouching, walking, running, shuffling, skipping,dancing, ascending or descending stairs, a ramp or a ladder, cycling,situps, crunches, pushups, skiing, snowboarding, lifting weights,deadlift, lunges, jumping jacks, squats, mountain climbers, bench dips,step ups, planking, calf raises, burpees, growing, shoulder press, bicepcurls, tricep dips, speed-walking, skateboarding, bowling, driving,tapping feet or otherwise fidgeting, or other steps or footfalls of thesubject, which may change speed, pace, or direction, etc.). In suchcases, heel pressure sensors on either foot may expect to have anegative correlation to one another during walking, as the heel of onefoot is in the swing phase while the other is striking the ground, or inthe stance phase. Conversely, the toes of the foot exert high and lowpressures at similar times, and so sensors near these toes arepositively correlated. Also in such cases, as the subject takes a slowstep over about five seconds, in the first two to three seconds,readings on a heel may have a similar amount of pressure as at the startof the step. Later in the step cycle, the amount of pressure on the heelmay be less than at the start of the step. Characterizing step data inthis manner may be facilitated by pressure sensors that measure anamount of pressure. Other pressure sensors may sense only whether apreselected pressure threshold has been exceeded, which may conservebandwidth compared with receiving data of an amount of pressure.

Connections between nodes are made through functions specific to eachnodal connection relationship. The activity classification processbegins with the data matrix of the input data being divided into subsetsof a defined time window duration. Weights and biases are set based onany known correlations, which may be from the training databaseincluding previous empirical data, simulated data, and synthesized databased on empirical data. Each time window of the input data is passedthrough the NN. The time window may be a time window, in which a windowof defined time duration is progressed across the data matrix. Only dataconstrained by the time bounds of the time window has the NN applied toit. After application of the NN, and subsequent classification, the timewindow is progressed along the time axis of the input data, and theinput data confined within the new time-bounds of the time window issubjected to the NN.

The time window may apply a weighting to a portion of the input datawithin the time window. Weighting of the input data will highlight aportion of the input data within the time window. A known and previouslyapplied event detection method may define an event window. Such an eventdetection method may identify events of interest within the input data,for example, a step within a walking session. The event window is thenbe defined as the input data within the bounds of the event window thatthe event detection method delineates. The event window subset of datais the subjected to the NN, and the input data within the event windowis classified.

The NN includes multiple layers. Where a CNN is applied, the CNNincludes at least one convolutional layer and one fully-connected layer.Each convolutional layer includes a convolutional step in which one ormore filter masks are applied to each of one or more feature maps havingactivation functions for establishing connections with nodes in asubsequent layer, and a pooling step. The filter masks may be updated asthe masks are applied to different portions of the data matrix todifferently account for the pressure data depending on the portion ofthe data matrix to which the filter mask is applied. The filter masksmay also be updated in the convolutional layers depending on the portionof a feature map to which the filter mask is applied.

Connections between nodes in adjacent layers may be as described in Eq.1.y _(j) ^(k)=max_(l∈C) _(j) {ƒ((Σ_(h=0) ^(M)Σ_(i∈D) _(l) w _(il) ^(hk) x_(i))+b _(j) ^(k))}  (Eq. 1)

In Eq. 1, a connection between a value x_(i) at node i to a value y_(j)^(k) at a related node j is established. Node i is in a first layer andnode j is in an adjacent second layer, with index k for the feature mapsof the adjacent second layer. The first layer may be an input datamatrix or convolutional layer. The second layer may be a convolutionallayer or a fully connected layer.

The value x_(i), is multiplied by the weight, w_(il) ^(hk), of thecorresponding relationship between the related nodes i and j. Relatednodes are included in subset D_(i). Subset D_(i) represents nodes thatare spatially close, logically close or temporally close. Sensors thatare logically close may include sensors that have an expectedrelationship or correlation to one another. These products are summedtogether. The summation of the products, and inclusion of only a definedsubset of values represents the convolution process.

This process is performed a number of times in order to increase therobustness of the system. Each nodal connection between has its ownweights and biases. There are M feature maps in this layer, indexed byh. A bias b_(j) ^(k), specific to the relationship between each node, isadded to the summation. Where the first layer is an input data layer,there are M data matrices in the input layer.

The calculated summations are applied in a pre-determined activationfunction f. The rectified linear unit function relu(x) may be theactivation function between convolutional layers. The results are pooledto reduce the number of nodes that pass from the first layer to thesecond layer. One approach to pooling is maximum pooling, as in Eq. 1.In maximum pooling, the largest node within a subset C_(j) is the onlyvalue to pass through to the next layer. The convolution steps andpooling steps are performed in each convolutional layer.

The convolution step and the pooling step may be applied to the matrixof the input data or to a feature map of the convoluted data using afilter mask that accounts for data from one sensor over time. Whenapplied to the input data, the convolution and pooling steps mayassociate time nodes corresponding to the same sensor at differentpoints in time, and which are temporally close to one another, such aswithin a predetermined timespan before and after the instancerepresented by the node. When applied to the input data, the convolutionstep and the pooling step may associate nodes corresponding to multiplesensors at a single moment in time. The associations may result inweights and biases that correspond to positive, negative, ornon-existent correlations between sensor data.

The filter masks may also be updated for different portions of a datamatrix or feature map to which the filter masks are applied. In somecases, a filter mask may apply non-zero values to time points for afirst sensor, but at a first selected certain time point, apply non-zerovalues to a first group of related sensors at the first selected timepoint. Similarly, the filter mask may apply non-zero values to a secondgroup of related sensors at a second selected time point. In such cases,the filters may assign a non-zero value to all time data in each sensor,and at certain time points for each sensor, assign a non-zero value toother sensors' readings at the certain time points.

After processing in the convolutional layers, the values are processedthrough one or more fully connected layers to provide classifiedactivity data. As the name may suggest, nodes in fully connected layersare connected to every other node in subsequent layers. No convolutionor pooling steps are employed. An example of a function for a fullyconnected layer is shown in Eq. 2:

$\begin{matrix}{y_{j} = {g\left( {\left( {\sum\limits_{i = 1}^{N}{w_{ij}x_{i}}} \right) + b_{j}} \right)}} & {{Eq}.2}\end{matrix}$

In the fully connected layers, the values x_(i) and y_(j) of connectednodes i and j are similarly subject to weights and biases. The weightw_(ij) is multiplied by the value of x_(i), the products are similarlysummed together and added to a bias b_(j), and this value is placedthrough another activation function f. The hyperbolic tangent functionmay be used as an activation function for the fully connected layers.

The data set in a fully connected layer is a one-dimensional vectorrather than a two-dimensional matrix as in the convolutional layers.Additional data, such as specific attributes of the subject, may beconcatenated to the vector of the first fully connected layer. The inputdata following the convolutional layers and the concatenated data arepassed through the first fully connected layer and any subsequent fullyconnected layers. The output from the final fully connected layer isclassified activity data.

The classified activity data may be presented as a set of percentagesthat represent the likelihood that the activity matches each of a set ofpossible activities. For example, the system may record a long sessionwhere a subject engages in several activity types. The resultingclassified activity data for a given subset of the input data within thesession may describe an estimation that the activity has a highlikelihood of being walking. The classified activity data for a secondgiven subset of the data may describe a high likelihood that the inputdata in that subset describes running. Transitions between activitiesmay be characterized as a combination of two activities, withlikelihoods of being one or the other. A person switching from walkingto sprinting may have a transition period where a window reports anoutput score comprised partially of walking and partially of running.Transitions between phases are classified into classified activity datathat include non-binary outputs, with the non-binary outputs defining aratio of one activity to another activity in the classified activitydata (e.g. jogging could be classified as partially running andpartially walking, skipping could be classified as partially running andpartially jumping, etc.).

Initially, the weights and biases between each node are set randomly oraccording to previously-characterized activity. The weights and biasesfor a given nodal relationship may be assigned values within a rangethat corresponds to the anticipated relationships between nodes. In oneexample, pressure sensors on opposite heels may have a negativecorrelation with one another for characterizing a walking activity, andas such, the random weight and bias values may be centered around anormalized large negative number in order to ensure a closer fit to anegative correlation when training the NN. These weights are notdefinite, so if these assumptions made regarding the sensorrelationships prove to be incorrect, the NN will resolve the assumption.Setting the weights and biases arbitrarily will allow the NN to definethese parameters on its own without applying any assumptions.

Supervised Training of the NN includes a forward step and abackpropagation step. For a CNN, in the forward step, data is propagatedthrough the convolutional layers (Eq. 1) and fully connected layers (Eq.2) to classify the data into activities, providing predicted activitydata. A similar approach is taken for a NN other than a CNN with respectto the layers of the NN. The predicted activity data is compared tocontrol activity data, and the difference between the two is defined asloss function. The loss function may be the difference, the squareddifference or the cross entropy between the predicted activity data andthe control activity data. The backpropagation step then uses this errorto adjust the weights and biases. The adjustments are computed using thegradients of the values and the weights and biases, through the inverseof the activation function related to the index of the node from whichthe pooling step (e.g. maximum values, minimum values, mean values,etc.) was taken. Dropout, noise, or other regularization techniques maybe used to prevent overfitting the experimental data. A derivative ateach node may be applied to a loss function to minimize the lossfunction and define adjustments to the weights and biases to mitigatethe error and train the NN. The control activity data may be providedfrom a confirmation input that the subject completed a defined activityor from a training confirmation received from a training databaseincluding data of known activities.

A system may be individualized to a specific subject by highlightinginput data collected from that subject in a subject training process.Subject training includes providing the input data to the NN while thesubject performs a defined activity and providing the confirmation inputto the NN that the defined activity has been performed. The NNclassifies activities based on the input data to provide the classifiedactivity data. The defined activity is known and the correspondingclassified actual activity data that is the expected classificationresult from application of the NN to the input data associated with thedefined activity is known. The classified actual activity data iscompared with the classified activity data. The NN may be updatedaccording to a loss function defined between the classified activitydata and the classified actual activity data by adjusting the weightsand biases to minimize the loss function. The subject's specificattributes direct the input data, and as a result, subject training isbased on data that is inherently specific to the subject. Subjecttraining may be applied following a prompt to the subject to perform thedefined activity or during the ordinary course of activityclassification. The confirmation input may be from a user verifying thatthe subject performed or is about to perform the defined activity. Whencombined with population training, classified activity data from subjecttraining may be weighted and biased accordingly and added to thetraining database.

Subject training may be used in combination with population training totrain a NN. The weights and biases within a subset of layers of the NNmay be locked following population training, leaving only a separatesubset of weights and biases open to change during further training.During subsequent subject training, only the weights and biases withinthose layers that are not locked may be modified according to the lossfunction.

Population training includes providing the training data to the NN. Thetraining data may include pressure data and other input data acquiredfrom a variety of subjects with varying specific attributes, performinga variety of known activities. The NN classifies activities based on thetraining data to provide classified training data. The actual activitiesand the corresponding classified actual activity data are known. Theclassified actual activity data is compared with the classified trainingdata. The NN may be updated according to a loss function defined betweenthe classified training data and the classified actual activity data byadjusting the weights and biases to minimize the loss function. Storedclassified activity data may be combined with the training data andincluded in the training database for facilitating population trainingfor other subjects. Training the system with the training data mayfacilitate application of the NN to classify activity for a new subject.

The training database may include empirical data and synthesized data.Empirical data may be acquired from a diverse variety of subjectsperforming a diverse variety of activities. To reduce the number ofactual subjects needed to train the system, synthesized data may becreated by randomly varying the empirical data. Varying the data seriesto provide synthesized data may be through random time shifting, randommagnitude scaling, random pattern changing or other approaches tovarying a dataset. Random time shifting uses a smoothed series of randomtime translations applied to each time data point in a data set. Randommagnitude scaling works similarly, whereby a smoothed scaling value isapplied to each sensor data point. Random pattern changing isaccomplished by making minor perturbations to the spectral amplitudes ofthe data set. These perturbations may be made globally to the entiredata series (e.g. with a Fourier Transform, etc.), or locally (e.g. withthe S-Transform, etc.). Other approaches to synthesizing data fromempirical data may include making minor smoothing adjustments to thetime-axis, magnitude, and local and global frequency spectrum such thatsaid synthesized values simulate variations in performance of activitiesof the same type. Use of synthesized data may mitigate overfitting ofthe NN by non-diversified training data.

The synthesized training data may facilitate preparing a system for usein characterizing activities of a given subject with less empiricaldata. Deep learning networks typically rely on a large training databaseto train initial weights and biases of the NN. Application ofsynthesized training data may reduce the number of subject trainingsessions based on the confirmation input required before using thesystem for characterizing activity of a given subject. The synthesizeddata can balance the training data to introduce diversity, such that thedistribution of activity data is less biased toward the aggregate trendsof the particular individuals in the database. In some cases, thetraining database may display trends that are not representative for thesubject or for a broader population than that in the training database(e.g. the training database may have a disproportionately largerepresentation of heavy footed individuals, etc.). Large amounts oftraining data can be synthesized for any selected activity, based onempirical data of the selected activity to base the synthesized data on.Using the synthesized data may reduce the amount of empirical datanecessary to train a NN for use on a given subject. Using thesynthesized data may mitigate over-fitting of the NN.

Training the NN may include initial population training based ontraining data to initialize the weights and biases. In such cases,training the NN may also adjust the weights and biases based on subjecttraining, and adjusting the weights and biases based on the subjecttraining may precede applying the NN to pressure data outside thecontext of the subject training. The training data may be updated withreference to the outcome of the subject training.

Processing of the classified activity data through the NN, thepopulation training or the subject training may be applied afterinputting user data with an input module. The user data may definespecific attributes of the subject and otherwise relate to the subject.Where the subject is an individual, user data may include personalattributes of the subject that may affect the activity classificationoutput by the NN (e.g. weight, height, age, diabetes, physical injuries,arthritis, Parkinson's, any other physical or mental conditions etc.).The user data may be applied to selecting initial weights and biasesbased on the training data or other previous data. Population trainingor subject training may facilitate application of the NN for classifyingactivity performed by the subject over a session of a few seconds bytaking into account personal attributes specific of the subject. If thesubject changes activities during the session, the NN may classifymultiple activities types performed during the session. The NN may alsoclassify transitions from one activity type to the next, and betweenseparate occurrences of one activity type.

The method and system described herein may use pressure data alone ormay use pressure data and other input data.

The method and system described herein may be applied to classify fallpredictors, the risk of falling, limb strain related to falling, orcombinations thereof.

The method and system described herein may be used to determine optimalshoe fitting, automatic shoe fitting, automatic shoe lacing, adjustingorthotics, cast fitting, adjusting fit of devices applying pressure tothe body.

The method and system described herein may be used to classifydifferences in the same activity such as balance metrics, or centre ofpressure etc. Differences in the same activity may highlight differencesassociated with concussion.

The method and system described herein may be used to classify the riskfor foot ulceration.

The method and system described herein may be used to support diagnosisof illness or disease (such as spinal chord lesions or injury,osteoarthrosis, Parkinson's etc.) based on quality of movement.

The method and system described herein may be used in userrehabilitation such as from a stroke, a traumatic brain injury or fromneurological deficits.

The method and system described herein may be used as a tool to supportdiagnosis of neuropathy or quantification of disease based on gaitchanges.

The method and system described herein may be used as to supportdiagnosis of gait abnormalities and deviations attributed toneurological conditions such as hemiplegic, spastic diplegic,neuropathic, myopathic, Parkinsonian, choreiform, ataxic (cerebellar),and sensory gait.

The method and system described herein may be used support diagnosis ofgait abnormalities such as antalgic gait, Charlie Chaplin gait,circumduction gait, limping, Lateral sway, Step variability, Internalrotation, or external rotation.

The method and system described herein may be used support diagnosis ofa previous amputation such as transtibial, transfemoral, kneedisarticulation, hip disarticulation, transmetatarsal, ray amputation,Chopart, Lisfranc, or ankle disarticulation.

The method and system described herein may be used to classify orpredict energy expenditure based on gait and movement.

The method and system described herein may be used to classify orpredict blood glucose, insulin requirements or combinations thereofbased on gait and movement of the user. Classification of how orthoticsaffect gait, how orthotics strain the sensors, to monitor orthotics forinitial fit or for ongoing assessment of fit may be facilitated by themethod and system described herein. The method and system describedherein may also be used for other equipment such as braces, retainers,prosthetics, prosthetic joints, slings, casts, or other medicalappliances that may apply a pressure to the user or appliances that theuser may apply pressure to, monitoring initial fit, monitoring ongoingcompliance of fit, monitoring degradation, monitoring activity postimplantation or combinations thereof.

The method and system described herein may be used for ergonomics suchas adjusting seat fit in a vehicle, workstation fit, equipment fitting,military gear fitting, helmet fitting, bra fitting.

The method and system described herein may be used to classifyimpairment such as for use in police service testing, occupationalhealth and safety, fatigue monitoring, long haul trucking testing andpilot testing.

The method and system described herein may be used for adjusting seatfit such as adjusting the seat in a vehicle, fitting the seat forvarious users, identifying the person in the seat, adjusting the fitbefore, during, or after driving, automatically adjusting the fit to thebody pressures of a user or combinations thereof. The classification canbe used for fit to car seats, bike seats, office chairs, wheelchairs,hospital beds, or any chairs or beds that a user applies pressure towhen sitting in or lying on.

The method and system described herein may be used for classificationfor user identification based on fingerprint, palm print, handprint,foot print, face print, walking across a mat, pressure signature on anybody part, pressure signature from any body part or combinationsthereof. The identification can be individual signature identificationor identification of subsets such as gender, weight category, or otherdemographic attributes.

The method and system described herein may be used for training andcoaching of physical activities to give the user feedback. Feedback maybe given on form, weight distribution, centre of gravity, handplacement, etc. Feedback may be given in real time or post-activity.Applications may include golf swing improvement with sensors in shoes,gloves or handle, with video or combinations thereof. Feedback may bereviewed by an expert, may include a web portal for an expert toclassify data to create a training data set or combinations thereof.

The method and system described herein may be used for classifyingefficiency of physical activities, determining efficiency rating,measuring rhythm, pace, and explosiveness, to predict a subject'sperformance on the activity.

The method and system described herein may be used for learningactivities based on the subject's pressures, physical activity inputsand combinations thereof.

The method and system described herein may be used for automaticadjustment of vehicle parts such as tire valve cap, tire pressure, wheelrotations, etc. based on pressure sensor data.

The method and system described herein may be used in robotic surgeryfor determination of anatomical structure based on mechanical propertiesas determined by pressure sensors on instrument tips or other sensors orcombinations thereof.

FIG. 1 shows a schematic of an activity classification system 10 forclassifying input data 20 including pressure data resulting fromactivities performed by a subject. A sensor module 11 and an inputmodule 16 receive the input data 20 and communicate the input data 20 toa processor 18. The processor 18 applies a deep learning neural networkthat is a convolutional neural network 30 (“CNN”) to the input data 20for classifying the subject's activity and resulting in classifiedactivity data 38 that may be expressed in any suitable form. A varietyof approaches may be used in presenting the classified activity data.Softmax is a function that normalizes probabilities to a range between 0and 1, and may be used in the final layer of a NN for finalclassification and allows multi-possibility classification rather thanbinary. For example, with Softmax, probabilities of 0.8 running, 0.2walking could be obtained rather than just 1.0 running, 0.0 walking.Other than Softmax, a sigmoid function, which would be binary only, orany other suitable normalization functions may be applied to present theclassified activity data 38.

The input data 20 may also be applied to training the CNN 30. Theprocessor 18 communicates the classified activity data 38 to an activitydata module 40. The activity data module 40 may include a communicationmodule 42 for communicating the classified activity data 38 to a user ora storage module 44 for storing the classified activity data 38. Theinput data 20 may be provided to the processor 30 through a temporarystorage module prior to any processing (e.g. a transitory local computerreadable medium, a transitory cloud-based storage, etc.).

The sensor module 11 includes a first pressure sensor 12 and a secondpressure sensor 14 for receiving sensor data. The first pressure sensor12 and the second pressure sensor 14 are positioned on a support matrix15. The first pressure sensor 12 receives first pressure data 22. Thesecond pressure sensor 14 receives second pressure data 24. A system mayinclude more than two pressure sensors (e.g. the system 110, etc.) oronly the first pressure sensor (e.g. the system 210, etc.). The firstpressure sensor 12, the second pressure sensor 14, and the input module16 may provide input data 20 to the processor 18. The first pressuresensor 12 and the second pressure sensor 14 may be combined with othersensors, as may be suitable for the application in respect of which thesystem 10 is applied (e.g. an accelerometer, a gyroscope, a seismograph,a thermometer, or a humidity sensor, an altimeter, a GPS, a videocamera, a heart rate sensor, an oxygen sensor, a breathing rate sensor,a blood glucose sensor, a fatigue measuring device, a limb positionmeasuring device, a blood pressure monitor, an ECG, a lung functionmeter, an alcohol level sensor, drug level sensor, or any other type ofsensor that measures the level of impairment, etc.). The first pressuresensor 12 or the second pressure sensor 14 may be either binary pressuresensors with a defined threshold for a positive reading, or may besensors that detect a pressure or force value.

The first pressure sensor 12 and the second pressure sensor 14 may belocated on the support matrix 15 at corresponding locations formeasuring pressure at a location on the subject's body and resultingfrom activities selected consistent with a particular application (e.g.at portions of an insole corresponding to a ball of a foot, portions ofan insole corresponding to a heel of a foot, portions of an insolecorresponding to a ball of a foot and a heel of a foot, fingertips of aglove, fingers of a glove, palm of a glove, elbow pads, knee pads,etc.). The first pressure data 22 and the second pressure data 24include data of measured pressure at the locations of the first pressuresensor 12 and the second pressure sensor 14 when worn by the subject.

The input module 16 is for providing user commands and user data 26 tothe processor 18. The user data 26 may include specific attributes ofthe subject. Where the subject is an individual, the specific attributesmay include personal attributes such physical traits, mental traits, orother data related to the subject that are to applied by the CNN 30 indetermining the classified activity data 38. For example, factors suchas weight, height, age, any physical or mental conditions (e.g.diabetes, physical injuries, arthritis, Parkinson's, etc.), or otherfactors that may affect the activity classification output by the CNN30, may be included in the user data 26. The user data 26 may be appliedas part of the input data 20, or may be used downstream of classifiedactivity data 38 to define features calculated with reference to boththe classified activity data 38 and the user data 26 separately (e.g. asin the systems 410 and 510, etc.).

FIG. 2 shows a schematic of the activity classification system 10adapted for use on a sole of a subject individual's foot. For suchapplications, the support matrix 15 may be in the shape of an insole,the first pressure sensor 12 may be located on the support matrix 15 ata location corresponding to the ball of the foot, and the secondpressure sensor 14 may be located on the support matrix 15 at a locationcorresponding to the heel of the foot. Functions provided by the inputmodule 16, the processor 18, and the activity data module 40, may beexecuted on or accessed through a first device 17 with reference tolocally stored or remotely accessed data and processes. The first device17 in the system 10 as shown in FIG. 2 is exemplified by a smartphone,but any suitable mobile or other device may be applied (e.g. smartphone,smartwatch, tablet, laptop computer, desktop computer, cloud-basedserver etc.). The sensor module 11 may include an inertial sensorconnected with a dorsal surface of the subject's shoe (not shown; anexample of a system with a sensor other than a pressure sensor is thesystem 410, which includes to the other sensor 409, which in the system410 is an inertial sensor). The first pressure data 22 and the secondpressure data 24 are transmitted from the sensor module 11 to the firstdevice 17 by a wireless transmitter 27.

The processor 18 is configured for executing instructions for applyingthe CNN 30 to the input data 20. The CNN 30 provides an activityclassification process, resulting in the classified activity data 38.The processor 18 may be located on any suitable mobile or other device(e.g. smartphone, smartwatch, tablet, laptop computer, desktop computer,cloud-based server etc.). Communication between the processor 18 and thefirst pressure sensor 12, the second pressure sensor 14, and the inputmodule 16, may be through any wired or wireless connection. In FIG. 2 ,the first pressure data 22 and the second pressure data 24 are shownbeing communicated to the first device 17 through a wireless connection.The processor 18 may be a graphics processing unit, a standard centralprocessing unit, or any suitable processing unit.

Returning to FIG. 1 , the CNN 30 includes at least one convolutionallayer 32 and at least one fully connected layer 34. The input data 20 isreceived by the processor 18. The input data 20 may include trainingdata 28. The training data 28 may be accessed from a training database29 accessed through a wired connection, through a wireless connection,from a remote service, or by any suitable access method. The trainingdata 28 is received by the processor 18 with the input data 20 and maybe provided to the CNN 30 to train the CNN 30. The training data 28 mayinclude empirical pressure data from a plurality of other subjects,including subjects having similar or different specific attributescompared with the current subject as determined with reference to theuser data 26, the first pressure data 22, the second pressure data 24,or any data received by the sensor module 11.

The convolutional layer 32 may be applied to the first sensor data 22,the second sensor data 24 and the user data 26 with reference to weightsand biases 36. The training data 28 may include empirical pressure dataof the other subjects each performing a plurality of activities. Theconvolutional layer 32 may be applied to the training data 28 and theuser data 26 with reference to the weights and biases 36. The weightsand biases 36 may be determined and updated with reference to the inputdata 20, including the first sensor data 22, the second sensor data 24,the user data 26, the training data 28 or the classified activity data38. The input data 20 is convoluted and pooled in the convolutionallayer 32, then provided to the fully connected layer 34, resulting inthe classified activity data 38.

A confirmation input 47 may be received from the input module 16 by theprocessor 18 and applied to train the CNN 30 based on classifiedactivity data 38 resulting from the first pressure data 22 and thesecond pressure data 24 acquired from the first pressure sensor 22 andthe second pressure sensor 24. Similarly, a training confirmation 45 maybe received from the training database 29 by the processor 18 andapplied to train the CNN 30 based on classified activity data 38resulting from the training data 28 received from the training database29. Either of the confirmation input 47 or the training confirmation 45may provide actual classified activity data for comparing with theclassified activity data 38 and training the CNN 30. The classifiedactivity data 38 may be applied to train the CNN 30 by updating theweights and biases 36 as part of training the CNN 30. The classifiedactivity data 38 may be communicated to the activity data module 40 forfeedback to be provided to the user through the communication module 42based on the classified activity data 38. The activity data module 40may allow for further processing and review of the classified activitydata 38 through the storage module 44. The classified activity data 38may be provided to a database for accessing in subsequent rounds oftraining and as part of the training data 28. In some cases, thetraining may be carried out with reference to the training data 28 aspart of configuration or updating of the system 10, and some suchsystems may function without application of the subject training.

FIG. 3 is a schematic of the CNN 30 processing the input data 20 intothe classified activity data 38. The input data 20 is passed through twoconvolutional layers 32 and three fully-connected layers 34 duringactivity classification to provide the classified activity data 38. Thetwo convolutional layers 32 include a first convolutional layer 31 and asecond convolutional layer 33. The three fully-connected layers 34include a first fully-connected layer 35, a second fully-connected layer37, and a third fully-connected layer 39. Each of the convolutionallayers 32 and the fully-connected layers 34 includes nodes located oneach feature map of the convolutional layer 32 or the fully-connectedlayer 34.

Nodes each include a datum or a greater portion of the input data 20. Anode may be connected to one or more nodes in an immediately subsequentor precedent layer. Connections between nodes are defined by anactivation function and subject to the weights and biases 36 applicableto the convolutional step of a particular connection, including asdescribed above with reference to Eq. 1. The weights and biases 36determine the connections between the nodes in subsequent and precedentconvolutional layers 20 based on the input data 20 and the relationshipsbetween the nodes. The user data 26 may be provided to the processor 18for application in the CNN 30 to determine initial weights and biases36, and in turn the relationships between nodes in adjacent layers. Theweights and biases 36 may apply to relationships between nodes inadjacent convolutional layers 32, such as between the firstconvolutional layer 31 and the second convolutional layer 33. Theweights and biases 36 may apply to relationships between nodes in aconvolutional layer 32 and nodes in an adjacent fully connected layer34, such as between the second convolutional layer 33 and the firstfully connected layer 35. The weights and biases 36 may apply torelationships between nodes in an adjacent fully connected layers 34,such as between the first fully connected layer 35 and the second fullyconnected layer 37.

The input data 20 is presented as a data matrix 80, shown as a twodimensional data array. Each of the convolutional layers 32 may applyone or more filter masks to the data matrix 80. Each separate filtermask may apply separate weights and biases, and resulting in a featuremap for each filter mask and each set of weights and biases. The firstconvolutional layer 31 may include a first feature map 81 a, a secondfeature map 81 b, and a third feature map 81 c, which result fromapplying three separate filter masks to the data matrix 80. The secondconvolutional layer 33 may include a first feature map 83 a, a secondfeature map 83 b, a third feature map 83 c, a fourth feature map 83 d, afifth feature map 83 e and a sixth feature map 83 f, which result fromapplying filter masks to each of the first feature map 81 a, a secondfeature map 81 b, and a third feature map 81 c. Nodes of the input data20 may be defined on the data matrix 80.

The data matrix 80 may be represented as a matrix including a pluralityof 1-dimensional vectors of the input data 20 for the first pressuresensor 22 and the second pressure sensor 24, with each vectorrepresenting the recorded values over time from one of the sensors. Inthe CNN 30, the features maps 81 a, 81 b and 81 c of the firstconvolutional layer 31, and the feature maps 83 a, 83 b, 83 c, 83 d, 83e and 83 f of the second convolutional layer 33 are abstracted from therepresentation of sensor data as a one-dimensional vector in time, inthe data matrix 80.

The data in the fully connected layers 34 may be defined as a1-dimensional vector in which each value is linked to every value in asubsequent fully connected layer through an activation function thatincludes the weights and biases 36.

The CNN 30 may be applied to the input data 20 during a convolutionalstep 62. During the convolutional step 62, the values in the input data20 that meet the qualifications of an activation function for aconvolutional step are multiplied by a weight and summed with a bias.Following the convolutional step, a pooling step 64 is applied torecover a datum and some of its neighbours in the data matrix 80 forpromotion to the feature maps 81 a, 81 b and 81 c of the firstconvolutional layer 31. During the convolutional step 62, the data ismultiplied by a weight, amplifying the amount of data. Pooling dataduring the pooling step 64 lowers the resolution and amount of the datain the convolutional layer, preserving bandwidth with respect to thedata in the data matrix 80 following the convolutional step 62. Theconvolutional step 62 and the pooling step 64 may draw correlations overtime only and from one individual sensor per data matrix 80 of the inputdata 20, or may draw correlations across multiple sensors per datamatrix 80 of the input data 20.

The pooling step 64 may include max pooling, wherein the maximum valueof a node and its neighbours is promoted as a single or reduced datasampling of all values at and close to the node that were promotedthrough the activation function at the convolutional step 62. Thepooling step 64 may include minimum pooling, wherein the minimum valueof a node and its neighbours is promoted as a single or reduced datasampling of all values at and close to the node that were promotedthrough the activation function at the convolutional step 62. Thepooling step 64 may include mean pooling, wherein the mean value of anode and its neighbours is calculated and a resulting mean valuepromoted as a single or reduced data sampling of all values at and closeto the node that were promoted through the activation function at theconvolutional step 62.

Application of the convolutional step 62 and the pooling step 64 to theinput data 20 results in the first feature map 81 a, the second featuremap 81 b, and the third feature map 81 c of the first convolutionallayer 31. Subsequent convolutional and pooling steps 66 may be appliedto the feature maps of the convolutional layers 32. The subsequentconvolutional and pooling steps 66 may include first subsequentconvolutional and pooling steps 65 and second subsequent convolutionaland pooling steps 67.

Application of the first subsequent convolutional and pooling steps 65to the nodes in the first feature map 81 a, the second feature map 81 b,and the third feature map 81 c of the first convolutional layer 31results in the first feature map 83 a, the second feature map 83 b, thethird feature map 83 c, the fourth feature map 83 d, and the fifthfeature map 83 e of the second convolutional layer 33.

After the features maps of the convolutional layers 32 have beendefined, the input data 20 may be filtered through the fully-connectedlayers 34. Application of the second convolutional and pooling step 67to the nodes in the first feature map 83 a, the second feature map 83 b,the third feature map 83 c, the fourth feature map 83 d, and the fifthfeature map 83 e results in a convolutional portion 43 of the firstfully-connected layer 35. The user data 26 may be concatenated with theconvolutional portion 43 of the first fully-connected layer 35 during aconcatenation step 72, defining a subject attribute portion 41 of thefirst fully-connected layer 35.

All nodes of data on the first fully connected layer 35 may be expressedas a first one-dimensional vector, which may be processed through anactivation function of a first connection step 74, resulting in thesecond fully-connected layer 37. Similarly, all nodes of data on thesecond fully connected layer 37 may be expressed as a secondone-dimensional vector, which may be processed through an activationfunction of a second connection step 76, resulting in the thirdfully-connected layer 39. A vector of the third fully-connected layer 39may be expressed at the classified event data 38. The activationfunctions of the first connection step 74 and the second connection step76 are calculated based on the results of the feature maps prepared frommultiple convolution and pooling steps during the convolutional layers,and do not include pooling steps.

FIG. 4 is a schematic of a method of activity classification 50 usingthe CNN 30 and the system 10. A time window (see the time window 73 ofFIG. 5 ) of known duration is defined on a timeline of the input data20. The input data 20 within the time window is processed to provide theclassified activity data 38. Updating the time window 52 may includeinitially defining the time window on the input data 20 or progressingthe time window along the timeline of the classified activity data 38.Distinct portions of the classified activity data 38 timeline within thetime window may be overlapping in time. Convolutional filtering 60 andfully-connected filtering 70 are applied to the input data 20 within thetime window, resulting in the classified activity data 38. A trainingquery 49 determines whether to check the classified activity data 38against the confirmation input 47. Regardless of the outcome at thetraining query 49, the method 50 includes is communicating theclassified activity data to a user 54 or storing the classified activitydata 56. The confirmation input 47 may be applied to training based oninput data 20 that includes the first pressure data 22 or the secondpressure 24. The training confirmation 45 may be applied to trainingbased on input data that includes training data 28. Based on theclassified activity data 38 and the confirmation input 47 or thetraining confirmation 45, a loss function is calculated between theclassified and actual activities 58, which may be applied to settingweights and biases 59. After setting the weights and biases 59,additional input data 20 may be received and applied to theconvolutional filtering 60 and the fully-connected filtering 70.

The input data 20 may include the first pressure data 22, the secondpressure data 24, the user data 26, or the training data 28, which maybe represented as the data matrix 80. After updating the time window 52,the input data 20 is applied to the first convolutional step 62 and thefirst pooling step 64 of the first convolutional layer 31. Thesubsequent convolutional steps and pooling steps 66 of the secondconvolutional layer 33 and any subsequent convolutional layers are alsoapplied.

User input data 26 may be concatenated onto a fully-connected layer atthe concatenation step 72. Data in a feature map following theconvolutional filtering 60 may be applied to the first fully-connectedlayer 35, which includes the user input data 26, and to any subsequentfully-connected steps 76 of the fully-connected filtering 70. Thefully-connected filtering 70 results in the classified activity data 38,which may be communicated to a user 54 or stored 56. The error betweenthe actual and classified activities may be calculated though subjecttraining with reference to the classified activity data 38 based on thefirst pressure data 22, the second pressure data 24 and any additionalempirical data obtained during the moving-window subset of time. Theerror between the actual and classified activities may be calculatedthough population training with reference to the classified activitydata 38 based on the training data 28. The training may be applied tosetting the weights and biases 59.

FIG. 5 shows input data 20 that is pressure data and a time windowlocated over the pressure data to defined a portion of the pressure datato which the NN will be applied.

FIG. 6 is a schematic of a system 110 for activity classification.

FIG. 7 is a schematic of a CNN 130 processing the input data 120 intothe classified activity data 138.

The system 110 includes the first pressure sensor 112, the secondpressure sensor 114, and a third pressure sensor 113. The wirelesstransmitter 127 transmits the first pressure data 122, the secondpressure data 124, and third pressure data 123 to the first device 117,shown as a laptop computer, for processing by the processor 118 byapplication of the CNN 130. The first device includes the communicationmodule 142 and the storage module 144. The training data (not shown) andtraining database (not shown) may be stored directly on the storagemodule 144 for local access without remote access, as with the trainingdatabase 29 of the system 10. The system 110 may also be used withoutany training database, wherein the supervised training is entirely usertraining.

The CNN 130 includes a single convolutional layer 132 and a singlefully-connected layer 134. The convolutional step 162 with threedifferent filter masks, and the pooling step 164 are applied to the datamatrix 180 of the input data 120, resulting in the first feature map 181a, the second feature map 181 b, and the third feature map 181 c of theconvolutional layer 132. The subsequent convolutional and pooling steps166 are applied to the first feature map 181 a, the second feature map181 b, and the third feature map 181 c of the convolutional layer 132,providing the first fully-connected layer 134. The first connection step176 is applied to the fully-connected layer 134, resulting in theclassified activity data 138.

The increased number of pressure sensors on the system 110 compared withthe system 10 may facilitate data classification with fewerconvolutional and pooling steps, and fewer fully-connected steps, in thesystem 110 compared with the system 10.

FIG. 8 is a schematic of a system 210 for activity classification. Thesensory module 211 includes only the first sensor 212. The first device217 includes the functions of the input module 216, and is shown as afirst smartphone. The first pressure data 222 is transmitted from thesensor module 211 to a second device 219, shown as a laptop computer.The second device includes the processor 218 and the storage module 244.The user data 226 is transmitted to the second device 219 by the firstdevice 217. The second device 219 accesses the training database 229 forproviding the training data 228 to the second device 219. The classifiedactivity data 238 may be stored on the storage module 244 and alsotransmitted to a third device 221, shown as a smartphone, which mayprovide the function of the communication module 242.

FIG. 9 shows a schematic of an activity classification system 310 forclassifying the input data 320 including the pressure data 322 resultingfrom activities performed by the subject. The first pressure sensor 312,an other sensor 309 and the training database 329 generate the inputdata 320. The input data 320 is communicated to the processor 318. Theprocessor 318 applies a deep learning neural network (“NN”) 390 to theinput data 320 for classifying the subject's activity and resulting inclassified activity data 338 that may be expressed in any suitable form.The input data 320 may also be applied to training the NN 390. Theprocessor 318 communicates the classified activity data 338 to theactivity data module 340. The activity data module 340 may include thecommunication module 342 for communicating the classified activity data338 to a user or the storage module 344 for storing the classifiedactivity data 338. The input data 320 may be provided to the processor318 through a temporary storage module prior to any processing (e.g. atransitory local computer readable medium, a transitory cloud-basedstorage, etc.).

The sensors include the pressure sensor 312 and the other sensor 309 forreceiving the input data 320. The other sensor 309 may include anysuitable sensor that measures data indicative of physical activity orother relevant factors (e.g. an accelerometer, a gyroscope, aseismograph, a thermometer, or a humidity sensor, an altimeter, a GPS, avideo camera, a heart rate sensor, an oxygen sensor, a breathing ratesensor, a blood glucose sensor, a fatigue measuring device, a limbposition measuring device, a blood pressure monitor, an ECG, a lungfunction meter, an alcohol level sensor, drug level sensor, or any othertype of sensor that measures the level of impairment, etc.). Thepressure sensor 312 generates pressure data 322. The other sensor 309receives other sensor data 325. The pressure sensor 312 and the othersensor 309 may be combined with other sensors, as may be suitable forthe application in respect of which the system 310 is applied. Thepressure sensor 312 or the other sensor 309 may be either binary sensorswith a defined threshold for a positive reading, or may be sensors thatdetect a quantitative value.

The input data 320 is received by the processor 318. The input data 320may include the training data 328. The training data 328 may be accessedfrom the training database 329 accessed through a wired connection,through a wireless connection, from a remote service, or by any suitableaccess method. The training data 328 is received by the processor 318with the input data 320 and may be provided to the NN 390 to train theNN 390. The training data 328 may include empirical pressure or otherdata from a plurality of other subjects, including subjects havingsimilar or different specific attributes compared with the currentsubject. The training data 328 may include synthetic data generated fromempirical pressure or other data from a plurality of other subjects,including subjects having similar or different specific attributescompared with the current subject. The training data 328 may includesimulated pressure data or other data.

The training confirmation 345 may be received from the training database329 by the processor 318 and applied to train the NN 390 based on theclassified activity data 338 resulting from the training data 328received from the training database 329. The training confirmation 345may provide actual classified activity data for comparing with theclassified activity data 338 and training the NN 390.

The classified activity data 338 may be communicated to the activitydata module 340 for feedback to be provided to the user through thecommunication module 342 based on the classified activity data 338. Theactivity data module 340 may allow for further processing and review ofthe classified activity data 338 through the storage module 344. Theclassified activity data 338 may be provided to the training database329 for accessing in subsequent rounds of training and as part of thetraining data 328. In some cases, the training may be carried out withreference to the training data 328 as part of configuration or updatingof the system 310, and some such systems may function withoutapplication of the subject training.

FIG. 10 is a flowchart of a method 350 for activity classification usingthe system 310. A weighted time window (weighted time window 357 shownin FIG. 11 ) is defined for the input data 320. The weighted time window357 is of a defined duration in time, and certain aspects of the inputdata 320 within the weighted time window 357 are assigned greatersignificance when being processed by the NN 390. The portions of theinput data 320 within the weighted time window 357 as it is progressedacross a timeline of the input data 320 may overlap. The method 310includes updating the weighted time window 351. Updating the weightedtime window 351 may include defining the weighted time window 357 forthe input data 320, changing the weighting features of the weighted timewindow or progressing the weighted time window 357 along the timeline ofthe input data 320.

The NN 390 is applied to the input data 320, resulting in the classifiedactivity data 338. The training query 349 provides a decision point todetermine whether to check the classified activity data 338 against thetraining confirmation 345. Where no training is to take place based onthe classified activity data 338, the system 310 communicates theclassified activity data 338 to a user 354 or stores the classifiedactivity data 356.

Where the system 310 is applying the training data 328 as the input data320 and is training based on the resulting classified activity data 338and the training confirmation 345, the loss function is calculated 358.The loss function is based on the difference between the classifiedactivity data 338 of the training data 328 and the training confirmation345. The loss function may be applied to setting weights and biases 359.After setting the weights and biases 359, additional input data 320 maybe received and applied to the neural network 390.

FIG. 11 shows the input data 320 and the weighted time window 357superimposed over the input data 320 to defined a portion of the inputdata 320 to which the NN 390 will be applied to provide the classifiedactivity data 338. A weighted portion 361 of the input data 320 withinthe weighted time window 357 may be weighted more heavily than theremainder of the input data 320 within the weighted time window 357 whenapplying the NN 390 to the input data 320 after the input data 320 isfiltered through the weighted time window 357.

FIG. 12 is a schematic of an activity classification system 410 forclassifying the input data 420 resulting from activities performed bythe subject. The sensor module 411 and the training database 429 providethe input data 420. The input data 420 is processed by the processor 418on the first device 417, which is shown as a smart phone. The sensormodule 411 is located on the support matrix 415. The communicationmodule 442 communicates the classified activity data 438 (see FIG. 13 )and classified combined metabolic and activity data, such as the energyexpenditure data 487, to a user of the system 410, each of which mayalso be stored on the storage module 444. The user may be the subjectwearing the insole with the support matrix 415 or a differentindividual. A health information database 497 is accessible from thefirst device 417 and a biological feature sensor 408 (e.g. a bloodglucose monitor, a blood insulin monitor, a monitor for othermetabolites in blood, a monitor for skin detectable metabolites, etc.)is present on the second device 419.

The sensor module 411 includes the first pressure sensor 412, the secondpressure sensor 414 and the other sensor 409 on the support matrix 415for receiving the input data 420. The other sensor 409 may include anysuitable sensor that measures a level of physical activity or othervariable accounted for by the NN 490 (see FIG. 13 ). The first pressuresensor 412 generates the first pressure data 422. The second pressuresensor 414 generates the second pressure data 424. The other sensor 409,in this case an inertial measurement unit (“IMU”) receives the othersensor data 425, in this case shown as IMU data. The first pressuresensor 412, the second pressure sensor 414 and the other sensor 409 maybe combined with additional sensors, as may be suitable for theapplication in respect of which the system 410 is applied. The firstpressure sensor 412, the second pressure sensor 414 and the other sensor409 may each be either binary sensors with a defined threshold for apositive reading, or may be sensors that detect a quantitative value.

The second device 419, shown as a smartwatch, includes the biologicalfeature sensor 408. The biological feature sensor 408 receives insulinuse data 488 or blood glucose data 489. The insulin use data 488 and theblood glucose data 489 are provided from the second device 419 to thefirst device 417 and the classified activity data 438 may be weightedalongside the insulin use data 488 or blood glucose data 489 to definethe energy expenditure data 487. The health information database 497 mayinclude formal medical records, user-inputted health information, orother relevant user data 426 that may also be weighted alongside theclassified activity data 438. The input module 416 may be used torequisition updated user data 426 from the health information database497 or to otherwise input information or commands for application in thesystem 410.

The input data 420 may include the training data 428. The training data428 may be accessed from the training database 429 through a wiredconnection, through a wireless connection, from a remote service, or byany suitable access method. The training data 428 is received by theprocessor 418 with the input data 420 and may be provided to the NN 490to train the NN 490. The training data 428 may include empiricalpressure or other data from a plurality of other subjects, includingsubjects having similar or different specific attributes compared withthe current subject. The training data 428 may include synthetic datagenerated from empirical pressure or other data from a plurality ofother subjects, including subjects having similar or different specificattributes compared with the current subject. The training data 428 mayinclude simulated pressure or other data. The training data 428 may beused to update the weights and biases applied in the NN 490.

FIG. 13 is a schematic showing functionality of the activityclassification system 410 directed to energy expenditure determinationbased on the classified activity data 438 in combination with theinsulin use data 488, blood glucose data 489 and the user data 426 fromthe health database 497. The first pressure data 422, the secondpressure data 424 and the IMU data 425 is input into activityclassification network 490. The activity classification network 490 isapplied to classify the input data 420 into classified activity data438. A metabolic load neural network (“MLNN”) 491 is applied to theclassified activity 438 along with user data 426 including electronicmedical records sourced from the health information database 497,insulin use data 488 or blood glucose monitor data 489, to determine theenergy expenditure data 487. The energy expenditure data 487 is basedboth on the classified activity data 438, and on the insulin use data488, blood glucose data 489 and the user data 426 from the healthdatabase 497. The classified activity data 438 may be communicateddirectly to a user at the communication module 442 or stored in thestorage module 444, as classified activity data 438 alone or as part ofthe energy expenditure data 487, which may also be communicated directlyto a user at the communication module 442 or stored in the storagemodule 444.

FIG. 14 is a schematic of an activity classification system 510 forclassifying the input data 520 resulting from activities performed bythe subject. The sensor module 511 and the training database 529 providethe input data 520. The input data 520 is processed by the processor 518on the first device 517, which is shown as a smart phone. The sensormodule 511 is located on the support matrix 515. The communicationmodule 542 communicates the classified activity data 538 (see FIG. 15 )and the energy expenditure data 587 to a user of the system 510, each ofwhich may also be stored on the storage module 544. The user may be thesubject wearing the insole with the support matrix 515 or a differentindividual. A health information database 597 is accessible from thefirst device 517 and a biological feature sensor 508 is present on thesecond device 519. A drug delivery system 594 is also in communicationwith the first device 517 for receiving insulin requirement data 592.Where the insulin requirement data 592 meets a defined metric, the drugdelivery system 594 will administer insulin 594 to the subject. Thebiological feature sensor 508 and the drug delivery system 594 may besimilarly adapted to measure other biological features and provide othermedications or substances to the subject.

The sensor module 511 includes the first pressure sensor 512, the secondpressure sensor 514 and the other sensor 509 on the support matrix 515for receiving the input data 520. The other sensor 509 may include anysuitable other sensor that measures a level of physical activity orother variable accounted for by the NN 590 (see FIG. 15 ). The firstpressure sensor 512 generates the first pressure data 522. The secondpressure sensor 514 generates the second pressure data 524. The othersensor 509, in this case an IMU, receives the other sensor data 525, inthis case shown as IMU data. The first pressure sensor 512, the secondpressure sensor 514 and the other sensor 509 may be combined withadditional sensors, as may be suitable for the application in respect ofwhich the system 510 is applied. The first pressure sensor 512, thesecond pressure sensor 514 and the other sensor 509 may each be eitherbinary sensors with a defined threshold for a positive reading, or maybe sensors that detect a quantitative value.

The second device 519, shown as a smartwatch, includes the biologicalfeature sensor 508. The biological feature sensor 508 receives insulinuse data 588 or blood glucose data 589. The insulin use data 588 and theblood glucose data 589 are provided from the second device 519 to thefirst device 517 and the classified activity data 538 may be weightedalongside the insulin use data 588 or blood glucose data 589 to definethe energy expenditure data 587. The health information database 597 mayinclude formal medical records, user-inputted health information, orother relevant user data 526 that may also be weighted alongside theclassified activity data 538. The input module 516 may be used torequisition updated user data 526 from the health information database597 or to otherwise input information or commands for application in thesystem 510.

The input data 520 may include the training data 528. The training data528 may be accessed from the training database 529 through a wiredconnection, through a wireless connection, from a remote service, or byany suitable access method. The training data 528 is received by theprocessor 518 with the input data 520 and may be provided to the NN 590to train the NN 590. The training data 528 may include empiricalpressure or other data from a plurality of other subjects, includingsubjects having similar or different specific attributes compared withthe current subject. The training data 528 may include synthetic datagenerated from empirical pressure or other data from a plurality ofother subjects, including subjects having similar or different specificattributes compared with the current subject. The training data 528 mayinclude simulated pressure or other data. The training data 528 may beused to update the weights and biases applied in the NN 590.

FIG. 15 is a schematic showing functionality of the activityclassification system 510 directed to energy expenditure determinationand administration of insulin, based on the classified activity data 538in combination with the insulin use data 588, blood glucose data 589 andthe user data 526 from the health database 597. The first pressure data522, the second pressure data 524 and the IMU data 525 is input intoactivity classification network 590. The activity classification network590 is applied to classify the input data 520 into classified activitydata 538. The MLNN 591 is applied to the classified activity 538 alongwith user data 526 including electronic medical records sourced from thehealth information database 597, insulin use data 588 or blood glucosemonitor data 589, to determine the energy expenditure data 587. Theenergy expenditure data 587 is based both on the classified activitydata 538, and on the insulin use data 588, blood glucose data 589 andthe user data 526 from the health database 597. The classified activitydata 538 may be communicated directly to a user at the communicationmodule 542 or stored in the storage module 544, as classified activitydata 538 alone or as part of the energy expenditure data 587, which mayalso be communicated directly to a user at the communication module 542or stored in the storage module 544.

In addition to communicating the energy expenditure data 587 to a userthrough the communication module 542 and storing the energy expendituredata in the storage module 544, the energy expenditure data 587 ispassed to an insulin requirement calculator 555 to define the insulinrequirement data 592. The insulin requirement data 592 is passed to theinsulin delivery module 594. Depending on criteria established for thesubject, including the user data 526 from the health informationdatabase 597, and on the insulin requirement data 592, the system 510may cause the insulin delivery module 594 to administer insulin to thesubject, report the insulin requirement data 592 through thecommunication module 542 and record the insulin requirement data 592 inthe storage module 544.

FIG. 16 shows a schematic of an activity classification system 610 forclassifying the input data 620 resulting from activities performed bythe subject for mitigating or preventing falls. The sensor module 611and the training database 629 provide the input data 620. The input data620 is processed by the processor 618 on the first device 617, which isshown as a smart phone. The sensor module 611 is located on the supportmatrix 615. The communication module 642 communicates the classifiedactivity data 638 to a user of the system. The user may be the subjectwearing the insole with the support matrix 615 or a differentindividual. The communication module 642 may also communicate theclassified activity data 638 to the subject through an alert output 699,which may be located on a garment 698, in the event that the classifiedactivity data 638 is indicative of an imminent fall. The communicationmodule 642 may also communicate the classified activity data 638 to ahealth care provider 695 in the event that the classified activity data638 is indicative of a fall. The classified activity data 638 may alsobe stored on the storage module 644.

The sensor module 611 includes the first pressure sensor 612, the secondpressure sensor 614 and the other sensor 609 on the support matrix 615for receiving the input data 620. The other sensor 609 may include anyother sensor that measures a level of physical activity or othervariable accounted for by the NN 690 (see FIG. 17 ). The first pressuresensor 612 generates the first pressure data 622. The second pressuresensor 614 generates the second pressure data 624. The other sensor 609,in this case an IMU receives the other sensor data 625, in this caseshown as IMU data. The first pressure sensor 612, the second pressuresensor 614 and the other sensor 609 may be combined with additionalsensors, as may be suitable for the application in respect of which thesystem 610 is applied. The first pressure sensor 612, the secondpressure sensor 614 and the other sensor 609 may each be either binarysensors with a defined threshold for a positive reading, or may besensors that detect a quantitative value.

The input data 620 may include the training data 628. The training data628 may be accessed from the training database 629 through a wiredconnection, through a wireless connection, from a remote service, or byany suitable access method. The training data 628 is received by theprocessor 618 with the input data 620 and may be provided to the NN 690to train the NN 690. The training data 628 may include empiricalpressure or other data from a plurality of other subjects, includingsubjects having similar or different specific attributes compared withthe current subject. The training data 628 may include synthetic datagenerated from empirical pressure or other data from a plurality ofother subjects, including subjects having similar or different specificattributes compared with the current subject. The training data 628 mayinclude simulated pressure or other data. The training data 628 may beused to update the weights and biases applied in the NN 690.

FIG. 17 is a flowchart of a method 650 for activity classification usingthe system 610. The method 650 includes updating an event window 653. Anevent window 675 (FIG. 18 ) is defined for the input data 620. The eventwindow 675 is of a duration in time equal to an identified event fromwithin the input data 620. The event may be identified by any suitablemethod, including by Fourier transform, adaptive filtering, or anysuitable approach, including as described in WIPO (PCT) PatentApplication No. CA2018/050802 to Cheng et al.

Depending on how the event is defined, the portions of the input data620 within the event window 675 as it is progressed across a timeline ofthe input data 620 may overlap. Updating the event window 653 mayinclude defining the event window 675 for the input data 620, changingthe selection features for the event window 675 or progressing the eventwindow 675 along the timeline of the input data 620 to the nextidentified event. The event window 675 may be applied only to a singletype of event or may be applied to multiple distinct types of events.The events may be any event indicative of activities that the system 610is being used to measure.

After updating the event window 653, the NN 690 is applied to a portionof the input data 620 within the event window 675, resulting in theclassified activity data 638. Based on the classified activity data 638,where the fall prediction query 648 is negative, the method 650 willreturn to updating the event window 653. Based on the classifiedactivity data 638, where the fall prediction query 648 is positive, themethod 650 communicates the classified activity data to a user 654. Animminent fall alert may be communicated to the subject through anoticeable alert, such as an audio alarm, tactile stimulus or otheroutput that is recognizable as urgent and simple to interpret, andproviding the subject some warning that the system 610 has assessed ahigh likelihood of the subject falling. The method 650 may also includestoring the classified activity data 656.

Where the system 610 is applying the training data 628 as the input data620 and is training based on the resulting classified activity data 638and the training confirmation 645, the loss function is calculated 658.The loss function is based on the difference between the classifiedactivity data 638 of the training data 628 and the training confirmation645. The loss function may be applied to setting weights and biases 659.After setting the weights and biases 659, additional input data 620 maybe received and applied to the neural network 690.

FIG. 18 shows the input data 620 and the event window 675 superimposedover the input data 620 to defined a portion of the input data 620 towhich the NN 690 will be applied to provide the classified activity data638. The event window 675 is bounded by event limits 677 defined withinthe input data 620. The event limits 677 are defined when updating theevent window 653.

The method and system may be used in sequence, with the output from afirst activity classification NN used as input for a second activityclassification NN. Multiple methods and systems of activityclassification may be cascaded.

FIG. 19 shows a schematic of an activity classification system 710 forclassifying the input data 720 including the pressure data 722 aresulting from activities performed by the subject. The first pressuresensor 712 a, the other sensor 709 and the training database 729generate the input data 720. The input data 720 is communicated to theprocessor 718. The processor 718 applies the first NN 790 a to the inputdata 720 for classifying the subject's activity and resulting in firstclassified activity data 738 a that may be expressed in any suitableform. The input data 720 may also be applied to training the first NN790 a. The processor 718 communicates the first classified activity data738 a to the activity data module 740. The activity data module 740 mayinclude the communication module 742 for communicating the firstclassified activity data 738 a to a user or the storage module 744 forstoring the first classified activity data 738 a. The input data 720 maybe provided to the processor 718 through a temporary storage moduleprior to any processing (e.g. a transitory local computer readablemedium, a transitory cloud-based storage, etc.).

The sensors include the pressure sensor 712 a and the other sensor 709for receiving the input data 720. The other sensor 709 may include anysuitable sensor that measures a level of physical activity. The pressuresensor 712 a generates pressure data 722 a. The other sensor 709receives other sensor data 725. The pressure sensor 712 a and the othersensor 709 may be combined with other sensors, as may be suitable forthe application in respect of which the system 710 is applied. Thepressure sensor 712 a or the other sensor 709 may be either binarysensors with a defined threshold for a positive reading, or may besensors that detect a quantitative value.

The input data 720 is received by the processor 718. The input data 720may include the training data 728. The training data 728 may be accessedfrom the training database 729 accessed through a wired connection,through a wireless connection, from a remote service, or by any suitableaccess method. The training data 728 is received by the processor 718with the input data 720 and may be provided to the first NN 790 a totrain the first NN 790 a. The training data 728 may include empiricalpressure or other data from a plurality of other subjects, includingsubjects having similar or different specific attributes compared withthe current subject. The training data 728 may include synthetic datagenerated from empirical pressure or other data from a plurality ofother subjects, including subjects having similar or different specificattributes compared with the current subject. The training data 728 mayinclude simulated pressure data or other data. The first classifiedactivity data 738 a and the second classified activity data 738 b mayalso be provided to the training database 729 for increasing the amountof training data 728 in the training database 729. The training data 728and the training confirmation 745 may be applied to train the second NN790 b.

The training confirmation 745 may be received from the training database729 by the processor 718 and applied to train the first NN 790 a basedon the first classified activity data 738 a resulting from the trainingdata 728 received from the training database 729. The trainingconfirmation 745 may provide actual classified activity data forcomparing with the first classified activity data 738 a and training thefirst NN 790 a. The training data 728 and the training confirmation 745may also be applied to train the second NN 790 b.

The first classified activity data 738 a may be communicated to theactivity data module 740 for feedback to be provided to the user throughthe communication module 742 based on the first classified activity data738 a. The activity data module 740 may allow for further processing andreview of the first classified activity data 738 a through the storagemodule 744. The first classified activity data 738 a may be provided tothe training database 729 for accessing in subsequent rounds of trainingand as part of the training data 728. In some cases, the training may becarried out with reference to the training data 728 as part ofconfiguration or updating of the system 710, and some such systems mayfunction without application of the subject training.

The first classified activity data 738 a is also provided to a second NN790 b. The second NN 790 b is applied to the first classified activitydata 738 a and the second pressure data 724. The second pressure sensor714 receives the second pressure data 724 for providing to the second NN790 b. The activity data module 740 may include the communication module742 for communicating the second classified activity data 738 b to auser or the storage module 744 for storing the second classifiedactivity data 738 b. The second classified activity data 738 b may alsobe provided to the training database 729 to supplement the training data728. The first pressure data 722 and the other data 725 may also beprovided to the second NN 790 b for classification into the secondclassified activity data 738 b.

In the preceding description, for purposes of explanation, numerousdetails are set forth in order to provide a thorough understanding ofthe embodiments. However, it will be apparent to one skilled in the artthat these specific details are not required. In other instances,well-known electrical structures and circuits are shown in block diagramform in order not to obscure the understanding. For example, specificdetails are not provided as to whether the embodiments described hereinare implemented as a software routine, hardware circuit, firmware, or acombination thereof.

Embodiments of the disclosure can be represented as a computer programproduct stored in a machine-readable medium (also referred to as acomputer-readable medium, a processor-readable medium, or a computerusable medium having a computer-readable program code embodied therein).The machine-readable medium can be any suitable tangible, non-transitorymedium, including magnetic, optical, or electrical storage mediumincluding a diskette, compact disk read only memory (CD-ROM), memorydevice (volatile or non-volatile), or similar storage mechanism. Themachine-readable medium can contain various sets of instructions, codesequences, configuration information, or other data, which, whenexecuted, cause a processor to perform steps in a method according to anembodiment of the disclosure. Those of ordinary skill in the art willappreciate that other instructions and operations necessary to implementthe described implementations can also be stored on the machine-readablemedium. The instructions stored on the machine-readable medium can beexecuted by a processor or other suitable processing device, and caninterface with circuitry to perform the described tasks.

The above-described embodiments are intended to be examples only.Alterations, modifications and variations can be effected to theparticular embodiments by those of skill in the art. The scope of theclaims should not be limited by the particular embodiments set forthherein, but should be construed in a manner consistent with thespecification as a whole.

What is claimed is:
 1. A method for determining energy expenditure dataof a subject comprising: generating a first set of input data of thesubject resulting from an activity, the first set of input datacomprising pressure data; applying a first deep learning neural networkto the first set of input data based on a first set of weights andbiases, resulting in classified activity data; training the first deeplearning neural network for updating the first set of weights andbiases; and applying a second deep learning neural network to a secondset of input data comprising the classified activity data based on asecond set of weights and biases, resulting in energy expenditure data.2. The method of claim 1, further comprising applying a time window tothe first set of input data to provide time-segmented input data; andwherein applying the first deep learning neural network to the first setof input data comprises applying the first deep learning neural networkto the time-segmented input data.
 3. The method of claim 1, whereintraining the first deep learning neural network comprises: confirmingthe activity, resulting in a defined activity and correspondingclassified actual activity data, wherein confirming the activitycomprises: prompting the subject to perform the defined activity;receiving a confirmation input that the subject performed the definedactivity; defining a loss function between the classified actualactivity data and the classified activity data; and updating the firstset of weights and biases for mitigating the loss function.
 4. Themethod of claim 1, wherein the first set of input data and/or the secondset of input data further comprises personal and/or physical attributesof the subject; and the personal and/or physical attributes of thesubject include health information from a database and/or self-inputtedhealth information.
 5. The method of claim 1, wherein the second set ofinput data further comprises data from a biological feature sensor. 6.The method of claim 1, wherein the energy expenditure data are passed toa drug requirement calculator to define a drug requirement data of adrug for the subject.
 7. The method of claim 6, wherein the drugrequirement data is used to alert the subject to self-administer a drugvia a visual alert, an audible alert, or a tactile alert.
 8. The methodof claim 6, wherein the drug requirement data is passed to a drugdelivery system; the drug requirement data meets a defined drugrequirement metric for the subject; and a dose of the drug isadministered to the subject through the drug delivery system.
 9. Themethod of claim 8, wherein the drug requirement data are insulinrequirement data; the drug delivery system is an insulin deliverysystem; and the drug is insulin.
 10. A system for determining energyexpenditure data of a subject comprising: a sensor module comprising apressure sensor, the sensor module for generating a first set of inputdata during an activity, the first set of input data comprising pressuredata; a processor included in a first device, configured for receivingthe first set of input data and for executing a method comprising:applying a first deep learning neural network to the first set of inputdata based on a first set of weights and biases, resulting in classifiedactivity data; training the first deep learning neural network forupdating the first set of weights and biases; and applying a second deeplearning neural network to a second set of input data comprising theclassified activity data based on a second set of weights and biases,resulting in energy expenditure data.
 11. The system of claim 10,wherein the sensor module is located on a support matrix; wherein thesupport matrix is located for receiving the first set of input data fromdifferent portions of the subject; and the first set of weights andbiases are initially determined with reference to the location of thesensor module on the support matrix.
 12. The system of claim 10, whereinthe first device is a smartphone, a smartwatch, a tablet, a laptopcomputer, a desktop computer, or a cloud-based server.
 13. The system ofclaim 10, further comprising a biological feature sensor for generatingbiological feature sensor data, and wherein the second set of input datafurther comprises the biological feature sensor data.
 14. The system ofclaim 13, further comprising a second device, and wherein the biologicalfeature sensor is included in the second device.
 15. The system of claim13, wherein the biological feature sensor is a blood glucose monitor, ablood insulin monitor, a monitor for other metabolites in blood, or amonitor for skin detectable metabolites.
 16. The system of claim 10,wherein the processor is further configured to communicate the energyexpenditure data to a drug requirement calculator to define a drugrequirement data for the subject.
 17. The system of claim 16, whereinthe first device is in communication with a drug delivery systemcontaining a drug; the drug requirement data meets a defined drugrequirement metric for the subject; and the drug delivery system isconfigured to administer a dose of the drug to the subject.
 18. Thesystem of claim 17, wherein the drug requirement data are insulin drugrequirement data; the drug delivery system is an insulin deliverysystem; and the drug is insulin.
 19. The system of claim 16, wherein thefirst device is further configured to communicate at least one of theclassified activity data, the energy expenditure data, and the drugrequirement data.
 20. The system of claim 16, wherein the first deviceis further configured to store at least one of the classified activitydata, the energy expenditure data, and the drug requirement data.